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Introduction : 

While  communication  and  teamwork  skills  are  increasingly  recognized  as  important  factors  in 
improving  patient  safety,  team  training  is  not  routinely  incorporated  into  training  programs, 
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are  developing  and  testing  a  screen-based  team  training  to  provide  healthcare  professionals 
deliberate  practice  on  teamwork  skills  and  improve  performance  through  automated, 
individualized  feedback. 

Methodology : 

During  the  past  year  (year  1)  we  reviewed  the  state  of  the  art  regarding  team  training  and 
identified  gaps  in  knowledge  and  practices  through  literature  review,  subject  matter  expert 
(SME)  interviews  and  focus  group  facilitation.  Available  screen-based  training  modules  and 
games  were  identified  and  reviewed. 

Results : 

Over  1500  articles  were  reviewed,  5  SMEs  interviewed  and  4  interprofessional  focus  groups 
conducted.  Interviews  revealed  a  degree  of  concordance  between  the  views  of  experts  and  those 
in  the  field:  teamwork  is  valuable  and  valued,  albeit  not  meeting  expectations  in  many 
instances.  Providing  context,  repeated  practice  and  feedback  are  important  aspects  of 
implementing  teamwork  skills.  The  screen-based  game  format  will  provide  these  elements. 
Conclusions : 

Single  player  games  may  not  be  suitable  for  teaching  all  teamwork  skills;  however,  teaching 
every  skill  is  less  important  than  deliberate  practice  implementing  selected  skills  in 
context . 
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1.  INTRODUCTION:  Narrative  that  briefly  (one  paragraph)  describes  the  subject,  purpose  and 
scope  of  the  research. 

While  communication  and  teamwork  skills  are  increasingly  recognized  as  important  factors  in 
improving  patient  safety,  team  training  is  not  routinely  incorporated  into  graduate  training  or 
continuing  medical  education  programs.  Opportunities  to  practice  teamwork  skills  and  receive 
objective  feedback  are  limited.  We  are  developing  and  testing  a  screen-based  team  training  to 
provide  healthcare  professionals  deliberate  practice  on  teamwork  skills  and  improve  performance 
through  automated,  individualized  feedback. 


2.  KEYWORDS:  Provide  a  brief  list  of  keywords  (limit  to  20  words). 

Teamwork  training,  assessment,  screen-based  simulation,  communication,  leadership,  situation 
monitoring,  mutual  support,  psychological  safety 

3.  ACCOMPLISHMENTS:  The  PI  is  reminded  that  the  recipient  organization  is  required  to 
obtain  prior  written  approval  from  the  awarding  agency  Grants  Officer  whenever  there  are 
significant  changes  in  the  project  or  its  direction. 

What  were  the  major  goals  of  the  project? 

List  the  major  goals  of  the  project  as  stated  in  the  approved  SOW.  If  the  application  listed 
milestones/target  dates  for  important  activities  or  phases  of  the  project,  identify  these  dates  and 
show  actual  completion  dates  or  the  percentage  of  completion. 


❖  Major  Task  1.  Identify  current  team  training  practices,  performance  gaps,  and  resources 
(originally  planned  mos.  1-6,  for  revised  completion  based  upon  6-month  no-cost  extension  see 
below). 

❖  Major  Task  2.  Create  domain  ontology  and  scenario  scripts  (originally  planned  mos.  5-8;  for 
revised  completion  based  upon  6-month  no-cost  extension  see  below). 

❖  Major  Task  3.  Design  a  framework  for  online  team  training  and  assessment  (originally  planned 
mos.  8-11;  for  revised  completion  based  upon  6-month  no-cost  extension  see  below). 

♦t*  Major  Task  4.  Build  the  screen-based  simulation  {Evaluation  and  Game-Play  Modes) 

(originally  planned  mos.  8-13;  for  revised  completion  based  upon  6-month  no-cost  extension 
see  below). 

*t*  Major  Task  5.  Conduct  research  using  the  screen-based  simulation  (originally  planned  mos.  6- 
8,  14-24;  for  revised  completion  based  upon  6-month  no-cost  extension  see  below). 


What  was  accomplished  under  these  goals? 

For  this  reporting  period  describe:  1 )  major  activities;  2)  specific  objectives;  3)  significant 
results  or  key  outcomes,  including  major  findings,  developments,  or  conclusions  (both  positive 
and  negative);  and/or  4)  other  achievements.  Include  a  discussion  of  stated  goals  not  met. 
Description  shall  include  pertinent  data  and  graphs  in  sufficient  detail  to  explain  any  significant 
results  achieved.  A  succinct  description  of  the  methodology  used  shall  be  provided.  As  the 
project  progresses  to  completion,  the  emphasis  in  reporting  in  this  section  should  shift  from 
reporting  activities  to  reporting  accomplishments. 
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For  the  purposes  of  this  report: 

Sim  Team=Randolph  Steadman,  Yue  Ming  Huang,  Rukhsana  Khan  and  Maria  Rudolph 

Education  Team=Noreen  Webb,  Federica  Raia,  Rachel  Lewin  and  Michael  Smith 

CASrr  (Center  for  Advanced  Surgical  &  Interventional  Technology)  Team=Areti  Tillou  and  Yen-Yi 

Juo 

CRESST  (Center  for  Research  on  Evaluation,  Standards  &  Student  Testing)  Team=Alan  Koenig,  John 
Fee,  Markus  Iseli  and  Charles  Parks 

❖  Major  Task  1.  Identify  team  training  practices,  performance  gaps  and  resources  (mos.  I- 14). 

>  Subtask  I :  Review  research  on  existing  team  training  literature,  available  products  and  best 

practices:  100%  completed. 

■  The  purpose  of  this  review  was  to  examine  existing  literature,  research,  products,  software, 
and  tools  to  identify  current  team  training  practices  and  performance  gaps  during  high  stakes 
medical  team  activities.  The  review  focused  on  areas  of  research  that  would  be  helpful  for 
curricular  framework  and  design  of  this  project’s  screen-based  simulation  for  training  and 
automated  assessment  of  teamwork  skills. 

■  We  focused  on  the  following  topics:  teamwork  dimensions  for  team  training,  effectiveness  of 
team  training  in  healthcare,  measuring  acquisition  of  teamwork  knowledge  and  skills,  impact 
of  teamwork  training  on  teamwork  knowledge  and  skills,  teamwork  attributes  that  are 
challenging  to  represent  in  a  single  player  screen-based  healthcare  simulation,  methods  of 
simulation-based  teamwork  training,  design  issues  related  to  authenticity  of  screen-based 
simulations,  feedback  and  debriefing  in  simulated  teamwork  settings,  screen-based 
simulations  versus  high  fidelity  simulators,  and  screen-based  simulations  teaching  teamwork 
in  medical  settings. 

■  Major  findings  for  this  review  will  be  summarized  in  Subtask  5,  along  with  a  description  of 
the  methodology  used. 

>  Subtask  2:  Perform  video  analysis  of  medical  teams  in  action:  100%  completed. 

■  Videos  of  critical  incidents  (real  and  simulated)  were  reviewed  by  all  members  of  the 
research  team. 

■  Videos  were  used  to  develop  consensus  of  what  the  observable,  assessable  teamwork  actions 
are  in  critical  care. 

■  Challenges  were  categorized  into  the  following  areas  of  opportunity  for  improvement:  a) 
communication  issues  as  they  relate  to  noise  control,  handoffs,  closed-loop  communication, 
leadership  and  anticipating/sharing  plan;  and  b)  process  issues  related  to  role  clarity  and 
delegation. 

>  Subtask  3:  Interview  Subject  Matter  Experts  (SMEs)  and  healthcare  practitioners.  100% 

completed. 

■  The  purpose  of  the  SME  interviews  was  to  gather  information  from  team  training  experts  to 
help  ascertain  the  strengths,  weaknesses  and  limitations  of  teamwork  training  approaches  and 
factors  that  contribute  to  the  breakdown  of  teamwork  processes.  The  research  team  also 
solicited  input  from  SMEs  on  the  proposed  team  training  framework  and  core  teamwork 
skills  that  will  be  incorporated  in  the  screen-based  simulation  game. 

■  The  interviews  were  conducted  in  a  semi- structured  format  and  led  by  Michael  Smith,  due  to 
his  expertise  in  applied  linguistics. 
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■  We  interviewed  the  following  team  training  subject  matter  experts:  David  Gaba,  MD, 
creator  of  Anesthesia  Crisis  Resource  Management;  David  Baker,  developer  of 
TeamSTEPPS;  Andrea  Amodeo,  TeamSTEPPS  researcher;  and  John  Holcomb,  MD,  a 
combat  surgery  expert.  Although  not  formally  named  as  an  SME  on  this  grant,  the  team 
also  interviewed  Christopher  Hund,  Director  of  Clinical  Quality  for  the  Health  Research  & 
Educational  Trust  (HRET)  due  to  his  expertise  in  directing  TeamSTEPPS  projects. 

■  Close  analysis  of  the  SME  interviews  helped  inform  the  research  team  on  the  following: 
which  context  is  teamwork  crucial,  what  skills  are  important  for  high  acuity  settings,  what 
undermines  teamwork,  traits  and  practices  for  a  “good”  team  player,  challenges  to  teaching 
teamwork,  advantages  and  pitfalls  of  TeamSTEPPS,  educational  tools  for  teaching 
teamwork  and  military  implications  that  affect  teamwork. 

■  Major  findings  from  SME  interviews  will  be  summarized  in  Subtask  5. 

>  Subtask  4:  Conduct/analyze  focus  groups  of  healthcare  teams:  100%  completed. 

■  The  purpose  of  the  focus  group  interviews  was  to  gather  honest  impressions  about 
teamwork  and  communication  from  healthcare  professionals  who  work  in  team  settings  at 
Ronald  Reagan  UCEA  Medical  Center.  The  intention  was  to  have  a  conversation  with 
front-line  medical  personnel  about  their  personal  experience  as  a  team  leader/  and  or 
supportive  team  member,  in  order  to  help  inform  our  game  development. 

■  Eour  focus  group  interviews  took  place  between  June  of  2017  and  July  of  2017.  All 
interviews  were  audio  recorded  with  consent  from  participants.  Rachel  Eewin  and  Maria 
Rudolph  conducted  the  interviews;  Ms.  Eewin  was  the  primary  interviewer,  while  Ms. 
Rudolph  took  notes.  Eocus  groups  were  comprised  of  3-6  front-line  medical  personnel, 
including  Emergency  Medicine  and  Internal  Medicine  physicians,  respiratory  therapists, 
and  nurses  from  the  ICU  and  Trauma  Surgery. 

■  A  few  recurrent  themes  were  gathered  from  the  interviews  including  introduction  styles 
from  team  members,  roles/structure  of  teams,  psychological  safety  to  encourage  feedback, 
and  clear  communication.  A  formal  narrative  of  findings  will  be  included  under  Subtask  5. 

>  Subtask  5:  Prepare  report  of  current  practices  and  gaps  in  team  training.  90%  completed. 

■  Information  gathered  from  SME  and  healthcare  team  interviews  is  being  compiled  into  the 
full  report.  We  anticipate  having  a  completed  report  in  month  14  (September  2017). 

❖  Major  Task  2.  Create  domain  ontology  and  scenario  scripts  (mos.5-21). 

>  Subtask  1:  Create  team  training  core  skills  domain  ontology:  100%  completed. 

■  Eollowing  completion  of  video  review,  the  Sim,  CASTT  and  CRESST  team  began  meeting 
weekly  to  create  a  list  of  all  possible  assessable  teamwork  actions  pertinent  to  patient  care. 
This  list  forms  a  pool  that  will  inform  the  assessable  actions  that  will  be  selected  for  use  in 
the  screen-based  simulation. 

■  A  final  version  of  the  domain  ontology  was  developed  based  on  the  assessable  teamwork 
skills  and  actions  that  were  identified.  Since  the  last  quarterly  report,  a  few  modifications 
regarding  the  relationship  between  certain  actions  and  teamwork  skills  have  been  made. 
See  file  “Team  Training  Ontology  v3”  in  Appendices  section. 
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>  Subtask  2:  Create  a  set  of  features,  affordances,  and  actions  for  user  interface:  100% 

completed. 

■  Using  the  game  development  software,  Unity,  an  early  test  environment  for  the  game  was 
developed  to  highlight  potential  interactivity  elements  (Parks). 

■  Acquired  3D  assets  (game  avatars)  including  hospital  room  and  medical  personnel 
(Huang). 

■  Created  a  test  environment  for  recognizing  speech  on  a  Macintosh  computer  (Koenig). 

■  Working  on  modes  of  interactivity  for  the  user  interface  (e.g.,  how  to  direct 
communication  to  the  desired  avatar). 

■  Currently  working  on  programming  a  mock-up  of  the  first  scenario  for  presentation  to  the 
research  team  for  feedback  on  interactivity  elements  and  sequencing  of  events  (Parks) 

■  Developed  an  Action  Level  Ontology  that  includes  possible  actions  and  the  related  game 
mechanics.  See  file  “Team  Training  Ontology  -  Action  Level  Ontology”  in  Appendices 
section  (Iseli,  Koenig,  &  Lee). 

>  Subtask  3:  Create  a  range  of  scenario  settings/events:  90%  completed. 

■  A  subset  of  the  research  team  composed  of  the  Sim  Team  and  CRESST,  met  to  establish 
the  learning  objectives,  setting,  sequence  of  events  and  player  affordances  for  the  first 
scenario  of  the  game. 

■  It  was  decided  that  the  first  scenario  would  focus  on  teaching  leadership  and 
communication  in  the  setting  of  the  trauma  bay  with  a  patient  involved  in  a  motor  vehicle 
accident  (MVA),  who  also  has  a  femur  fracture. 

■  An  inventory  of  skills  appropriate  for  the  scope  of  practice  for  the  roles  of  the  player  and 
non-player  characters  (NPCs)  for  this  first  scenario  was  also  created. 

>  Subtask  4:  Create  a  knowledge  assessment  (baseline  team  skills)  scenario:  75%  completed. 

■  The  baseline  scenario  (MVA)  will  serve  as  a  pretest  for  player/student  Knowledge,  Skills 
and  Abilities  (KSAs);  assessing  the  player’s  knowledge  and  skills  related  to  teamwork 
without  any  feedback  until  after  scenario  completion. 

>  Subtask  5:  Create  3  scenarios  with  different  settings,  events  and  skill  requirements:  30% 

completed. 

■  The  research  team  brainstormed  a  series  of  events  and  skill  requirements  to  be 
incorporated  in  the  game.  As  stated  under  Subtask  3  above,  the  learning  objectives,  setting 
and  sequence  of  events  for  the  first  scenario  of  the  game  has  been  created. 

■  The  team  is  currently  working  on  establishing  the  possible  branching  points  the  player 
may  take  should  they  not  follow  the  optimal  sequence  of  events  delineated. 

■  Subsequent  scenarios  are  currently  in  development,  and  are  following  the  same  model  and 
structure  used  for  this  first  scenario.  We  will  incorporate  the  information  gathered  from  the 
SME  and  focus  group  interviews  into  each  of  these  subsequent  scenarios. 

■  All  scenarios  include  the  same  basic  and  advanced  teamwork  skills,  including  conflict 
resolution  (under  leadership  and  mutual  support). 

❖  Major  Task  3.  Design  a  framework  for  online  team  training  and  assessment  (mos.  8-16). 

>  Subtask  1:  Design  the  automatic  assessment  engine:  80%  completed 

■  A  Bayesian  Network  (BN)  has  been  created  from  the  ontologies.  The  BN  is  used  to  infer 
competencies  related  to  the  teamwork  skill  constructs.  The  main  constructs  (top  level 
nodes)  include:  leadership,  situation  monitoring,  communication,  mutual  support,  and 
conflict  resolution.  At  the  observable  level,  there  are  the  types  of  actions  the  player  can 
take  and  the  components  of  each  action  that  the  simulation  captures  and  scores.  See  file 
named  “Bayesian  Network  of  Teamwork  Skills”  in  Appendices  section. 
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>  Subtask  2:  Design  the  simulation  interface:  80%  completed. 

■  Gameplay  mode:  The  simulation  interface  includes  several  affordances  through  which  the 
player  can  carry  out  various  actions,  along  with  a  text  input  for  spoken  dialogue  (note:  this 
will  incorporate  natural  language  processing  for  relating  text  to  game  actions) 

■  Evaluation  mode:  The  interface  will  look  mostly  similar  to  the  Gameplay  mode,  with 
differences  being  this  will  have  video  controls  to  play  and  pause  the  scenario,  along  with  an 
interface  for  the  player  to  provide  written  comments  about  noteworthy  teamwork  skills 
observed  (correctly  or  incorrectly). 

>  Subtask  3:  Design  the  after- action  reporting:  80%  completed. 

■  Player  performance  will  be  presented  and  explained,  including  evaluation  of  player  actions 
with  descriptive  feedback,  and  general  instruction  on  the  specific  team  skills  required  in  the 
game  scenario. 

>  Subtask  4:  Pilot  storyboard  workflows  for  quality  assurance  piloting:  75%  completed. 

■  Storyboarding  is  complete;  currently  vetting  with  medical  SMEs. 

❖  Major  Task  4:  Build  screen  based  simulation  (Evaluation  and  Gameplay  modes)  (mos.  8-21). 

>  Subtask  1:  Develop  software  specifications:  70%  complete. 

■  Agile  development  of  software  specifications  based  on  use-cases,  with  specific  associated 
development  sub-tasks.  Process  consists  of  Specify  ->  Develop  ->  Test  ->  Iterate/Refine, 
which  inter- links  Subtasks  1,  2,  and  3. 

>  Subtask  2:  Develop  software-based  prototype  of  two  simulation  modes:  40%  complete. 

■  Development  is  occurring  using  an  agile  development  methodology  (see  Subtask  1,  above). 

■  Due  to  unanticipated  personnel  shortfalls,  we  are  behind  schedule  on  the  programming.  To 
mitigate  the  risk  this  poses  to  our  schedule,  we  are  in  the  process  of  hiring  additional 
programming  help.  We  are  also  preparing  a  request  for  a  no-cost  extension  of  six  months. 

>  Subtask  3:  Perform  software  testing  for  quality  assurance:  40%  complete. 

■  Testing  is  occurring  using  an  agile  development  methodology  (see  Subtask  1,  above). 

*t*  Major  Task  5.  Conduct  research  using  the  screen-based  simulation  (mos.  6-8,  21-30). 

>  Subtask  1:  Obtain  IRB  approval  from  UCEA  and  USAMRMC  HRPO:  100%  completed. 

■  We  submitted  the  UCEA  IRB  application  in  December  2016  and  received  approval  on 
Eebruary  15,  2017.  We  also  submitted  a  Protocol  Submission  Eorm  for  USAMRMC  HRPO 
on  March  28,  2017  and  received  approval  on  May  1 1,  2017.  No  study  activities  have  taken 
place  yet. 
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What  opportunities  for  training  and  professional  development  has  the  project  provided? 

If  the  project  was  not  intended  to  provide  training  and  professional  development  opportunities  or 
there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 


Describe  opportunities  for  training  and  professional  development  provided  to  anyone  who 
worked  on  the  project  or  anyone  who  was  involved  in  the  activities  supported  by  the  project. 
“Training”  activities  are  those  in  which  individuals  with  advanced  professional  skills  and 
experience  assist  others  in  attaining  greater  proficiency.  Training  activities  may  include,  for 
example,  courses  or  one-on-one  work  with  a  mentor.  “Professional  development”  activities 
result  in  increased  knowledge  or  skill  in  one’s  area  of  expertise  and  may  include  workshops, 
conferences,  seminars,  study  groups,  and  individual  study.  Include  participation  in  conferences, 
workshops,  and  seminars  not  listed  under  major  activities. 


Nothing  to  Report. 


How  were  the  results  disseminated  to  communities  of  interest? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  the  results  were  disseminated  to  communities  of  interest.  Include  any  outreach 
activities  that  were  undertaken  to  reach  members  of  communities  who  are  not  usually  aware  of 
these  project  activities,  for  the  purpose  of  enhancing  public  understanding  and  increasing 
interest  in  learning  and  careers  in  science,  technology,  and  the  humanities. 

Nothing  to  Report. 


What  do  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals? 

If  this  is  the  final  report,  state  “Nothing  to  Report.  ” 


Describe  briefly  what  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals 
and  objectives. 


The  research  team  plans  to  continue  holding  weekly  full  team  meetings  to  meet  the  goals  of  the  project. 
A  subset  of  the  research  team  has  been  holding  separate  weekly  meetings  to  focus  on  game 
development  and  will  continue  to  hold  these  meetings  to  accomplish  this  goal.  We  are  also  in  the 
process  of  hiring  additional  programmers  (at  no  extra  cost)  to  advance  progress  in  building  the  screen- 
based  simulation. 
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4.  IMPACT:  Describe  distinctive  contributions,  major  accomplishments,  innovations,  successes,  or 
any  change  in  practice  or  behavior  that  has  come  about  as  a  result  of  the  project  relative  to: 

What  was  the  impact  on  the  development  of  the  principal  discipline(s)  of  the  project? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  findings,  results,  techniques  that  were  developed  or  extended,  or  other  products 
from  the  project  made  an  impact  or  are  likely  to  make  an  impact  on  the  base  of  knowledge, 
theory,  and  research  in  the  principal  disciplinary  field(s)  of  the  project.  Summarize  using 
language  that  an  intelligent  lay  audience  can  understand  (Scientific  American  style). 


Nothing  to  Report. 


What  was  the  impact  on  other  disciplines? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  the  findings,  results,  or  techniques  that  were  developed  or  improved,  or  other 
products  from  the  project  made  an  impact  or  are  likely  to  make  an  impact  on  other  disciplines. 


We  anticipate  that  this  study  will  inform  the  defense  community  and  private  sector  on  the  effectiveness  of 
screen -based  simulation  for  teamwork  skills  training  of  healthcare  providers.  We  also  hope  to  provide  the 
design  methodology  for  the  development  of  screen-based  simulation  training  on  other  topics  and  for  other 
disciplines. 


What  was  the  impact  on  technology  transfer? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  ways  in  which  the  project  made  an  impact,  or  is  likely  to  make  an  impact,  on 
commercial  technology  or  public  use,  including: 

•  transfer  of  results  to  entities  in  government  or  industry; 

•  instances  where  the  research  has  led  to  the  initiation  of  a  start-up  company;  or 

•  adoption  of  new  practices. 

Nothing  to  Report. 


11 


What  was  the  impact  on  society  beyond  science  and  technology? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  results  from  the  project  made  an  impact,  or  are  likely  to  make  an  impact,  beyond 
the  bounds  of  science,  engineering,  and  the  academic  world  on  areas  such  as: 

•  improving  public  knowledge,  attitudes,  skills,  and  abilities; 

•  changing  behavior,  practices,  decision  making,  policies  (including  regulatory  policies), 
or  social  actions;  or 

•  improving  social,  economic,  civic,  or  environmental  conditions. 


Nothing  to  Report. 


5.  CHANGES/PROBLEMS:  The  Project  Director/Principal  Investigator  (PD/PI)  is  reminded  that 
the  recipient  organization  is  required  to  obtain  prior  written  approval  from  the  awarding  agency 
Grants  Officer  whenever  there  are  significant  changes  in  the  project  or  its  direction.  If  not 
previously  reported  in  writing,  provide  the  following  additional  information  or  state,  “Nothing  to 
Report,”  if  applicable: 


Changes  in  approach  and  reasons  for  change 

Describe  any  changes  in  approach  during  the  reporting  period  and  reasons  for  these  changes. 
Remember  that  significant  changes  in  objectives  and  scope  require  prior  approval  of  the  agency. 


Nothing  to  Report. 


Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them 

Describe  problems  or  delays  encountered  during  the  reporting  period  and  actions  or  plans  to 
resolve  them. 


We  have  found  that  establishing  the  details  and  sequence  of  events  for  scenarios  has  taken 
longer  than  expected,  thus  have  tasked  a  subset  of  the  research  team  to  meet  weekly  to  make 
progress  in  this  area.  We  are  also  behind  in  game  programming  due  to  unanticipated  personnel 
shortcomings.  To  mitigate  the  risk  of  falling  behind  schedule,  we  are  in  the  process  of  hiring 
additional  programmers  (at  no  extra  cost).  Programming  funds  are  being  reallocated  to  result  in 
no  change  in  programming  costs.  One  programmer  is  already  in  position  and  the  source  for  a 
second  additional  programmer  has  been  identified.  The  second  programmer  will  be  onboarded 
in  the  next  4  weeks.  We  are  also  creating  a  programming  task  list  with  timeline  and 
dependencies  to  ensure  timely  completion  of  tasks.  Programmers  will  also  be  assigned  stand¬ 
alone  tasks  to  optimize  work.  Lastly,  we  are  in  the  process  of  preparing  a  6-month  no-cost 
extension  request  to  ensure  completion  of  the  project  and  have  modified  our  timeline 
accordingly. 
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Changes  that  had  a  significant  impact  on  expenditures 

Describe  changes  during  the  reporting  period  that  may  have  had  a  significant  impact  on 
expenditures,  for  example,  delays  in  hiring  staff  or  favorable  developments  that  enable  meeting 
objectives  at  less  cost  than  anticipated. 


Nothing  to  Report. 


Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards, 
and/or  select  agents 

Describe  significant  deviations,  unexpected  outcomes,  or  changes  in  approved  protocols  for  the 
use  or  care  of  human  subjects,  vertebrate  animals,  biohazards,  and/or  select  agents  during  the 
reporting  period.  If  required,  were  these  changes  approved  by  the  applicable  institution 
committee  (or  equivalent)  and  reported  to  the  agency?  Also  specify  the  applicable  Institutional 
Review  Board/Institutional  Animal  Care  and  Use  Committee  approval  dates. 

Significant  changes  in  use  or  care  of  human  subjects 


Nothing  to  Report. 


Significant  changes  in  use  or  care  of  vertebrate  animals. 


Nothing  to  Report 
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Significant  changes  in  use  of  biohazards  and/or  select  agents 


Nothing  to  Report 


6.  PRODUCTS:  List  any  products  resulting  from  the  project  during  the  reporting  period.  If 
there  is  nothing  to  report  under  a  particular  item,  state  “Nothing  to  Report.” 

•  Publications,  conference  papers,  and  presentations 

Report  only  the  major  publication(s)  resulting  from  the  work  under  this  award. 

Journal  publications.  List  peer-reviewed  articles  or  papers  appearing  in  scientific, 
technical,  or  professional  journals.  Identify  for  each  publication:  Author(s);  title; 
journal;  volume:  year;  page  numbers;  status  of  publication  (published;  accepted, 
awaiting  publication;  submitted,  under  review;  other);  acknowledgement  of  federal 
support  (yes/no). 


Nothing  to  Report. 


Books  or  other  non-periodical,  one-time  publications.  Report  any  book,  monograph, 
dissertation,  abstract,  or  the  like  published  as  or  in  a  separate  publication,  rather  than  a 
periodical  or  series.  Include  any  significant  publication  in  the  proceedings  of  a  one-time 
conference  or  in  the  report  of  a  one-time  study,  commission,  or  the  like.  Identify  for  each 
one-time  publication:  Author(s);  title;  editor;  title  of  collection,  if  applicable; 
bibliographic  information;  year;  type  of  publication  (e.g.,  book,  thesis  or  dissertation); 
status  of  publication  (published;  accepted,  awaiting  publication;  submitted,  under 
review;  other);  acknowledgement  of  federal  support  (yes/no). 


Abstract  submitted  on  8/24/2017  to  American  Educational  Research  Association  (AERA) 
2018  Annual  Meeting.  Currently  under  review  and  we  anticipate  status  of  acceptance  on 
1 1/2/2017.  See  file  “AGIEE  Methodology  for  Developing  a  Game-Based  Assessment  of 
Teamwork  Skills”  in  appendices  section  below. 

Authors:  Alan  Koenig,  PhD,  John  Eee,  PhD,  Markus  Iseli,  PhD  and  Randolph  Steadman, 
MD,  MS 

Title  of  Abstract:  AGIEE  Methodology  for  Developing  a  Game-Based  Assessment  of 
Teamwork  Skills 
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Other  publications,  conference  papers,  and  presentations.  Identify  any  other 
publications,  conference  papers  and/or  presentations  not  reported  above.  Specify  the 
status  of  the  publication  as  noted  above.  List  presentations  made  during  the  last  year 
(international,  national,  local  societies,  military  meetings,  etc.).  Use  an  asterisk  (*)  if 
presentation  produced  a  manuscript. 


Nothing  to  Report. 


•  Website(s)  or  other  Internet  site(s) 

List  the  URL  for  any  Internet  site(s)  that  disseminates  the  results  of  the  research 
activities.  A  short  description  of  each  site  should  be  provided.  It  is  not  necessary  to 
include  the  publications  already  specified  above  in  this  section. 


Nothing  to  Report. 


•  Technologies  or  techniques 

Identify  technologies  or  techniques  that  resulted  from  the  research  activities.  In  addition 
to  a  description  of  the  technologies  or  techniques,  describe  how  they  will  be  shared. 


Nothing  to  Report. 


•  Inventions,  patent  applications,  and/or  licenses 

Identify  inventions,  patent  applications  with  date,  and/or  licenses  that  have  resulted  from 
the  research.  State  whether  an  application  is  provisional  or  non-provisional  and  indicate 
the  application  number.  Submission  of  this  information  as  part  of  an  interim  research 
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performance  progress  report  is  not  a  substitute  for  any  other  invention  reporting 
required  under  the  terms  and  conditions  of  an  award. 


Nothing  to  Report. 


•  Other  Products 

Identify  any  other  reportable  outcomes  that  were  developed  under  this  project. 
Reportable  outcomes  are  defined  as  a  research  result  that  is  or  relates  to  a  product, 
scientific  advance,  or  research  tool  that  makes  a  meaningful  contribution  toward  the 
understanding,  prevention,  diagnosis,  prognosis,  treatment,  and/or  rehabilitation  of  a 
disease,  injury  or  condition,  or  to  improve  the  quality  of  life.  Examples  include: 

•  data  or  databases; 

•  biospecimen  collections; 

•  audio  or  video  products; 

•  software; 

•  models; 

•  educational  aids  or  curricula; 

•  instruments  or  equipment; 

•  research  material  (e.g.,  Germplasm;  cell  lines,  DNA  probes,  animal  models); 

•  clinical  interventions; 

•  new  business  creation;  and 

•  other. 


Nothing  to  Report. 


7.  PARTICIPANTS  &  OTHER  COLLABORATING  ORGANIZATIONS 
What  individuals  have  worked  on  the  project? 

Provide  the  following  information  for:  (1)  PDs/PIs;  and  (2)  each  person  who  has  worked  at  least 
one  person  month  per  year  on  the  project  during  the  reporting  period,  regardless  of  the  source 
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of  compensation  (a  person  month  equals  approximately  160  hours  of  effort).  If  information  is 
unchanged  from  a  previous  submission,  provide  the  name  only  and  indicate  “no  change.  ” 


Example: 

Name:  Mary  Smith 

Project  Role:  Graduate  Student 

Researcher  Identifier  (e.g.  ORCID  ID):  1234567 
Nearest  person  month  worked:  5 

Contribution  to  Project:  Ms.  Smith  has  performed  work  in  the  area  of 

combined  error-control  and  constrained  coding. 
Funding  Support:  The  Ford  Foundation  ( Complete  only  if  the  funding 

support  is  provided  from  other  than  this  award). 


Name:  Randolph  Steadman,  MD,  MS 
Project  Role:  Principal  Investigator 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  2.7 

Contribution  to  Project:  Dr.  Steadman  has  performed  work  in  the  area  of  literature  and  product  review, 
video  analysis.  He  serves  as  clinical  content  expert  for  development  of  game  scenarios.  He  has  also 
provided  direction  and  oversight  of  the  project  as  principal  investigator. 

Name:  Yue  Ming  Huang,  EdD,  MHS 
Project  Role:  Co-Investigator;  Project  Manager 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  1.8 

Contribution  to  Project:  Dr.  Huang  has  overseen  the  administration  and  management  of  the  project. 
She  has  also  performed  work  in  the  area  of  literature  and  product  review,  video  analysis  and  provided 
input  for  game  design  and  objectives. 

Name:  Rukhsana  Khan,  MPH 
Project  Role:  Research  Assistant 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  2.28 

Contribution  to  Project:  Ms.  Khan  has  provided  assistance  in  project  management.  She  has  also 
performed  work  in  the  area  of  literature  and  product  review,  video  analysis,  game  development  and 
completion  of  IRB  application. 

Name:  Noreen  Webb,  PhD 
Project  Role:  Co-Investigator 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  2.88 

Contribution  to  Project:  Dr.  Webb  has  performed  work  in  the  area  of  literature  and  product  review, 
video  analysis  and  provided  input  for  game  design  and  objectives.  She  has  also  helped  write  the 
literature  review  report. 
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Name:  Federica  Raia,  PhD 
Project  Role:  Co-Investigator 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  3.0 

Contribution  to  Project:  Dr.  Raia  has  performed  work  in  the  area  of  literature  and  product  review, 
video  analysis  and  provided  input  for  game  design  and  objectives.  She  has  also  helped  write  the 
literature  review  report. 

Name:  Michael  Smith 

Project  Role:  Graduate  Student  Researcher 

Researcher  Identifier:  N/A 

Nearest  person  month  worked:  3.0 

Contribution  to  Project:  Mr.  Smith  has  performed  work  in  the  area  of  literature  and  product  review, 
video  analysis  and  provided  input  for  game  design  and  objectives.  He  also  led  subject  matter 
expert  interviews  and  helped  write  a  summary  of  those  findings. 

Name:  Rachel  Lewin 

Project  Role:  Graduate  Student  Researcher 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  2.88 

Contribution  to  Project:  Ms.  Lewin  has  performed  work  in  the  area  of  literature  and  product 
review,  video  analysis  and  provided  input  for  game  design  and  objectives.  She  also  led  focus  group 
interviews  of  healthcare  teams  and  helped  summarize  those  findings. 

Name:  Markus  Iseli,  PhD 
Project  Role:  Co-Investigator 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  1.5 

Contribution  to  Project:  Mr.  Iseli  has  helped  plan  and  design  the  domain  ontology  and  screen- 
based  simulation  interface. 

Name:  Alan  Koenig,  PhD 
Project  Role:  Co-Investigator 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  2.15 

Contribution  to  Project:  Dr.  Koenig  has  provided  leadership  over  the  planning  and  design  of  the 
domain  ontology  and  screen-based  simulation  interface. 

Name:  John  Lee,  PhD 
Project  Role:  Co-Investigator 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  1.35 

Contribution  to  Project:  Mr.  Lee  has  managed  the  CRESST  team’s  deliverables  specifically 
pertaining  to  the  design  and  development  of  domain  ontology  and  screen-based  simulation 
interface.  He  also  contributed  to  the  completion  of  the  IRB  application. 
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Name:  Charles  Parks 
Project  Role:  Programmer 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  7.08 

Contribution  to  Project:  Mr.  Parks  has  performed  work  in  framework  design  and  programming 
test  environments  for  the  screen-based  simulation  game. 

Name:  Yen-Yi  Juo,  MD 
Project  Role:  Research  Fellow 
Researcher  Identifier:  N/A 
Nearest  person  month  worked:  2.04 

Contribution  to  Project:  Dr.  Juo  has  provided  clinical  direction  in  the  development  of  potential 
game  scenarios  and  its  objectives. 


Has  there  been  a  change  in  the  active  other  support  of  the  PD/PI(s)  or  senior/key  personnel 
since  the  last  reporting  period? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

If  the  active  support  has  changed  for  the  PD/PI(s)  or  senior/key  personnel,  then  describe  what 
the  change  has  been.  Changes  may  occur,  for  example,  if  a  previously  active  grant  has  closed 
and/or  if  a  previously  pending  grant  is  now  active.  Annotate  this  information  so  it  is  clear  what 
has  changed  from  the  previous  submission.  Submission  of  other  support  information  is  not 
necessary  for  pending  changes  or  for  changes  in  the  level  of  effort  for  active  support  reported 
previously.  The  awarding  agency  may  require  prior  written  approval  if  a  change  in  active  other 
support  significantly  impacts  the  effort  on  the  project  that  is  the  subject  of  the  project  report. 


Nothing  to  Report. 
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What  other  organizations  were  involved  as  partners? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  partner  organizations  -  academic  institutions,  other  nonprofits,  industrial  or 
commercial  firms,  state  or  local  governments,  schools  or  school  systems,  or  other  organizations 
(foreign  or  domestic)  -  that  were  involved  with  the  project.  Partner  organizations  may  have 
provided  financial  or  in-kind  support,  supplied  facilities  or  equipment,  collaborated  in  the 
research,  exchanged  personnel,  or  otherwise  contributed. 

Provide  the  following  information  for  each  partnership: 

Organization  Name: 

Location  of  Organization:  (if  foreign  location  list  country) 

Partner’s  contribution  to  the  project  ( identify  one  or  more ) 

•  Financial  support; 

•  In-kind  support  (e.g.,  partner  makes  software,  computers,  equipment,  etc., 
available  to  project  staff); 

•  Facilities  (e.g.,  project  staff  use  the  partner’ s  facilities  for  project  activities); 

•  Collaboration  (e.g.,  partner’s  staff  work  with  project  staff  on  the  project); 

•  Personnel  exchanges  (e.g.,  project  staff  and/or  partner’s  staff  use  each  other’s  facilities, 
work  at  each  other’s  site);  and 

•  Other. 


Nothing  to  Report. 


8.  SPECIAL  REPORTING  REQUIREMENTS 

COLLABORATIVE  AWARDS:  For  collaborative  awards,  independent  reports  are  required 
from  BOTH  the  Initiating  PI  and  the  Collaborating/Partnering  PI.  A  duplicative  report  is 
acceptable;  however,  tasks  shall  be  clearly  marked  with  the  responsible  PI  and  research  site.  A 
report  shall  be  submitted  to  https://ers.amedd.armv.niil  for  each  unique  award. 

QUAD  CHARTS:  If  applicable,  the  Quad  Chart  (available  on  https://www.usamraa.armv.mil) 
should  be  updated  and  submitted  with  attachments. 
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9.  APPENDICES:  Attach  all  appendices  that  contain  information  that  supplements,  clarifies  or 
supports  the  text.  Examples  include  original  copies  of  journal  articles,  reprints  of  manuscripts 
and  abstracts,  a  curriculum  vitae,  patent  applications,  study  questionnaires,  and  surveys,  etc. 


21 


DOD  Team  Training  Ontology 
(last  edited  8-2-2017) 


1.  Leadership 


Introduces  self  to  other  team 
members  {1a,  2a) 


Establishes  himself  or  herself  as 

the  leader:  "My  name  is _ ,  and  I  ^ 

will  be  assuming  leadership"  (1b) 


Elicits  or  knows  skills  of  various 
team  members  (1  a  &  1  c) 


Determines  what  skills/roles  are 
needed  {1c) 


Determines  if  more  or  less  team 
members  are  needed  (Iciv) 


Delegation:  Leader  assigns  tasks 
to  team  members  (Icii) 


Assesses  available  resources 
(equipment)  (lev) 


Recognize  need  for  leadership 
change  if  warranted  (Id) 


^  Manages  conflict  or  disagreements 
if  they  occur  (le) 


Shares  information  /mental  model 


through  briefings  /  huddles  (If) 

I 

Monitors  performance  and 
provides  constructive  feedback 

(ig) 

I  ~ 

Encourages  /  solicits  input  from 
other  team  members 

(ih) 


Gives  critical  instructions/orders  in 
a  timely  manner  (1  i,  2c) 


Queries  about  who 
has  experience 
dealing  with  problem 
at  hand 


Repeat  backs  instructions  /  orders  to 
ensure  closed  loop  communication 
(Checkbacks)  (2b) 


6.  Psychological 
Safety /Trust 


Responds  to 
challenges  with 
curiosity  versus  not 
interested  or  offended 
(5a) 


4.  Mutual  Support/ 
Conflict  Resolution 


Assisting  each  other  with 
high  workload  (4a) 


Request  assistance  from 
fellow  staff  when 
overwhelmed  (4b) 


Caution  each  other  about 
potentially  dangerous  •<- 
situations  (4c) 


Feedback  between  staff  is 
delivered  to  promote 
positive  interactions  {4d) 


□I 


Speaking  up:  staff 
advocates  for  patient  {4e) 


Speaking  up:  challenging 
others  {4f) 


Conveys  current  situation  to  a  new 
leader/member  as  applicable  (2d) 


Speaks  up  when  appropriate  (conveys 
disagreements/opinions)  (2e) 


Requests  additional  information  if 
needed  (2f) 


remains  quiet  when  needed{2g) 


3.  Situation 
Monitoring 


Maintain  Life  first  (3a) 


Time  pressures  (3b) 


Staff  anticipates  each  other's  needs  and 
what  will  be  needed  (3c) 

I  ~ 

Monitor  each  other's  performance  /  correct 
each  other's  mistakes  (3d) 

I 

Continuously  scan  the  environment  for 
important  information  (3e) 

I 

Share  information  regarding  changes  in 
patient  status,  other  team  members, 
environment,  progress  toward  plan 
(STEP)  (3f) 


Adapt  quickly  and  effectively  to  changes 
(3g) 


Conducts  task  in  the  right  order  /  stays 
on  task  {3h) 


Resolving  conflicts  (4g) 
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Team  Training  Ontology  -  Action  Level  Ontology 


ActionID 

Description:  Player... 

Category 

Available  Action 

Game  Mechanics 

PCintroducesSelf 

introduces  him/herself 

Supply  information 

Introduce  Self 

click  on  "address  all"  button,  free  text 

PCassumesLead 

assumes  leadership  role,  makes  their  role  clear 

Supply  information 

Assume  Leadership 

click  on  "address  all"  button,  free  text 

PCcedesLead 

relinquishes  leadership  role 

Supply  information 

Relinquish  Leadership 

click  on  "address  all"  button  or  on  NPC, 
free  text 

PCassignsTasksRoles 

assigns  tasks  to  team  member(s) 

click  on  NPC,  select  task  from  menu 
"do  X" 

PCreassignsTasksRoles 

PCperformsAssignedTask 

performs  task  that  has  been  assigned  to  him/her 

click  on  "do  myself  button,  select  task 
from  menu  "do  X" 

PCrequestsInfo 

requests  information  from  or  consults  with  people  (inside/outside  of  team, 
family  member)  or  other  resources  (e.g.  game) 

Request  information 

Get  Information 

click  on  NPC,  select  from  menu  "ask 
me"  or  "ask  me  face-to-face",  free  text 

PCprovidesInfo 

tell  team  members  about  status  of  patient/situation,  express  what  you  are 
thinking,  etc.  compare  with  PCbriefsNPCs 

Supply  information 

Provide  information 

click  on  "address  all"  button  or  on  NPC, 
free  text 

PCanswersQuestion  -> 
PCprovidesFeedback 

answers  questions  from  team  members 

Supply  information 

Answer  Questions 

PCprovidesFeedback 

gives  feedback  to  other  team  member{s)  (e.g.  "well  done!",  "next  time,  please 
make  sure  you  look  at  the  monitor  when  doing  ..."),  answers  questions, ... 

Supply  information 

Provide  Feedback 

PCencouragesFeedback 

elicicts  feedback 

Supply  information 

Encourage  Feedback 

PCfollowsUpOnIncompleteTasks 

follows  up  on  incomplete  tasks  (e.g.  by  reassigning  tasks,  doing  tasks 
him/herself,  or  taking  other  corrective  action) 

Request  information 

depends  on  next  action  chosen 

PCfollowsUpOnIncompleteInformation 

follows  up  on  incomplete  information  (e.g.  if  information  was  given  but 
incomplete) 

Request  information 

Clarification 

depends  on  next  action  chosen 

PCbriefsNPCs 

briefs  team  members  (e.g.  with  the  help  of  team  members,  player  assesses 
situation  and  makes  a  plan) 

Short  session  prior  to  start  to 
share  the  plan,  discuss  team  formation, 
assign  roles  and  responsibilities,  establish 
expectations  and  climate,  anticipate 
outcomes  and  likely  contingencies 

Supply  information 

Brief 

click  on  "address  all"  button,  select 
from  menu  "brief,  choose  text  from 
multiple-choice  menu 

PChuddlesWithNPCs 

huddles  with  team  members  (e.g.  player  initiates  information  sharing  with 
team  members  and  adapts  plan  if  necessary) 

Supply  information 

Huddle 

click  on  "address  all"  button,  select 
from  menu  "huddle",  choose  text  from 
multiple-choice  menu 

PCdebriefsNPCs 

debriefs  team  members  (e.g.  After  action  review:  PC  states/reflects  what 
went  well  and  what  could  be  improved) 

Informal  information  exchange 
session  designed  to  improve  team 
performance  and  effectiveness  through 
lessons  learned  and  reinforcement  of 
positive  behaviors 

Supply  information 

Debrief 

click  on  "address  all"  button,  select 
from  menu  "debrief,  enter  free  text 

PCobjects 

objects  (speaks  up)  to  other  team  members  (e.g.  to  prevent  a  possible 
mistake) 

Supply  information 

Object  /  Speak  up 

PCprovidesHelpToNPCs 

provides  help  to  other  team  members 

Supply  information 

Provide  help 

PCvoluntarilyPerformsTask 

performs  task  on  own  initiative  (e.g.  player  sees  that  nobody  was  assigned  to 
do  a  needed  task,  thus  player  performs  task  him/herself) 

click  on  "do  myself  button,  select  task 
from  menu  "do  X" 

PCpositionChange 

changes  his/her  position  in  room  (e.g.  to  get  perform  a  task,  to  have  a  better 
vantage  point,  to  make  room  for  other  team  members,  etc.) 

should  be  based  on  movement  in  the 
physical  space  (e.g.,  foot  of  bed,  or 
elsewhere) 

PCasksForAssistance 

requests  assistance  from  other  team  member 

Help  seeking 

PCasksLeaderForPermission 

if  PC  is  not  the  leader,  asking  the  leader  for  permission 

Request  information 

Ask  Leader  for  Permission 

These  player  actions  will  be  scored  on  a  selection  of  the  following  dimensions: 

appropriateness  (i.e.  was  the  action  required  or  would  another  action  have  been  a  better  choice?  was  the  action  helpful?) 
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Team  Training  Ontology  -  Action  Level  Ontology 


ActionID 

Description:  Player... 

Category 

Available  Action 

Game  Mechanics 

timeliness  (i.e.  was  the  timing  of  the  action  ok?) 

appropriate  addressee  (i.e.  if  the  action  was  related  to  communication  with  another  team  member:  Was  the  communication  addressed  at  the  appropriate  member  or  would  another  member  have  been  a  better  choice?) 

manner  (i.e.  if  the  action  was  related  to  communication  with  another  team  member:  Was  the  communication  made  in  a  polite,  respectful,  and/or  supportive  manner?  or  Did  the  player  make  eye-contact  with  the  addressee?) 

quality  (i.e.  was  information  complete, 
accurate,  was  action  done  vountary 
without  being  asked) 
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Bayesian  Network  of  Teamwork  Skills 
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AGILE  Methodology  for  Developing  a  Game-Based  Assessment  of  Teamwork  Skills 

Team  training  using  synthetic,  software-based  environments  -  whether  as  the  primary 
modality,  or  as  an  augment  to  in-person,  classroom-style  training  -  is  highly  desirable  to 
educators.  Indeed,  a  cornerstone  to  developing  effective  team  management  and  behavioral 
skills  is  having  the  ability  to  practice  those  skills  in  fidelity-relevant  settings  (McGaghie  et  al., 
2011).  However,  while  in-person,  role-playing  modes  of  pedagogy  can  be  beneficial,  they  are 
not  always  practical  nor  desirable,  as  limitations  on  environment,  resources,  personnel,  and 
other  deficiencies  to  authenticity  can  limit  the  validity  to  such  exercises.  In  contrast,  synthetic 
environments  in  the  form  of  computer-based  games  or  simulations  can  provide  robust, 
authentic  settings  in  which  to  teach,  practice,  and  assess  team  skills  using  scenario  features  & 
resources  deliberately  aligned  to  instructional  and  assessment  objectives  (Nyssen  et  al.,  2002; 
Schwid  et  al.,  2001;  Kreutzer  et  al.,  2016). 

But  while  the  benefits  of  using  simulation  to  teach  and  assess  team  skills  are  well  understood 
(Rosen  et  al.,  2008),  the  functional  implementation  of  a  synthetic  environment  tailored  to 
support  specific  team-training  pedagogy  is  not  a  trivial  undertaking.  Indeed,  not  only  must  the 
simulation  properly  represent  the  relevant  constructs  and  features  of  the  domain  (e.g.  an 
emergency  room  in  a  hospital  with  relevant  personnel,  equipment,  patient  information,  etc.), 
but  it  also  must  overlay  varied  and  modifiable  human  traits  and  behaviors  resident  within  each 
of  the  synthetic  team  members.  Furthermore,  the  training  and  assessment  arising  from  the 
simulation  must  be  able  to  distinguish  and  decouple  a  player's  domain  knowledge  &  skill  with 
their  skills  specifically  pertaining  to  teamwork  proficiency. 

In  this  paper,  we  describe  a  methodology  employed  in  the  design  and  development  of  just  such 
a  simulation  used  to  train  and  assess  medical  personnel  to  more  effectively  function  in  ad-hoc 
teams  in  critical-care  situations.  Although  this  effort  did  result  in  the  development  of 
sophisticated,  interactive  game-style  software,  this  paper's  focus  is  on  the  process  and 
methodology  employed  in  the  development,  rather  than  on  the  end-product  itself.  Specifically, 
we  present  a  process  that  addressed  competing  stakeholder  perspectives  and  requirements  - 
i.e.  that  of  educators  focused  on  instructional  and  assessment  design  of  team  training,  and  that 
of  medical  personnel  focused  on  performance  criteria  of  a  player's  (and  team's)  medical  actions 
and  decision-making. 

Like  with  the  development  of  so  many  pedagogical  games  and  simulations,  educators  can  find 
themselves  at  odds  with  the  subject  matter  experts  and  stakeholders  for  whom  they  are 
building  training.  To  address  this,  our  team  of  educators  and  medical  personnel  utilized  an 
AGILE-inspired  development  methodology  -  highly  iterative  in  nature,  breaking  tasks  into  small, 
chunk-sized  modules  that  could  be  built,  tested,  and  refined,  and  in  which  all  stakeholders 
maintained  an  active  involvement  from  the  project's  start  to  finish  (Douglass,  2016).  This  had 
the  benefit  of  not  only  teasing  out,  designing,  and  testing  ideas  in  rapid  fashion,  but  provided 
an  efficient  platform  for  incorporating  competing  stakeholder  requirements,  and  allowed 
unforeseen  issues  to  surface  early,  with  opportunities  for  low-cost  corrective  action.  This 
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approach  offers  utility  not  only  to  team  training  simulation,  but  to  any  pedagogical  simulation 
development. 
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Design  of  a  Screen-Based  Simulation  for  Training  and  Automated  Assessment  of  Teamwork  Skills 

Log  Number:  DM142062 

Award  Number:  W81XWH-1 6-1 -0308 

PI:  Randolph  Steadman,  MD  Org:  University  of  California,  Los  Angeles  Award  Amount:  $1 ,148,000 


Study/Product  Aim(s) 

•Aim  1 :  Identify  team  training  practices,  performance  gaps  and 
resources 

•Aim  2:  Create  domain  ontology  (necessary  actions,  decisions  and 

their  inter-relationships)  and  scenario  scripts 

•Aim  3:  Design  a  framework  for  online  team  training  and  assessment 

•Aim  4:  Build  screen-based  simulation  (evaluation  and  game-play 

modes) 

•Aim  5:  Conduct  research  study  using  the  screen-based  simulation 

Approach 

A  literature  review  will  identify  gaps  in  team  training,  limitations  of 
current  training  methods  and  essential  behavioral  skills.  This  phase 
will  identify  the  core  teamwork  skills  that  will  be  the  focus  of  the 
simulation-based  systems  we  develop.  We  will  then  create  the 
ontology  and  scenario  scripts  and  conceptualize  the  framework  for 
the  screen-based  simulation.  Finally,  we  will  evaluate  the  training 
effectiveness  and  user  interface  of  the  screen-based  simulation 
systems. 


Timeline  and  Cost 
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lA.  Review  existing  research,  products  and  best  practices 
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IS.  Perform  video  review  analysis  of  medical  teams  in  action 
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1C.  Interview  SMEs  and  healthcare  practitioners 
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ID.  Conduct/analyze  focus  groups  of  healthcare  teams 
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IE.  Prepare  report  of  current  practices  and  gaps  in  team  training  (Deliverable) 
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2.  Create  domain  ontology  and  scenario  scripts 
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2A.  Create  team  training  core  skills  domain  ontology 
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□ 

2B.  Create  set  of  features,  affordances  and  actions  for  user  interface 
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2C.  Create  range  of  scenario  settings/events 
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3.  Design  aframework  for  online  team  training  and  assessment 
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8-9 

8C% 

16(11/2017) 

; 

X 

3B.  Design  the  simulation  interface 
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4.  Build  screen-based  simulation  (Evaluation  and  Game-Pfay  modes) 
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4A.  Develop  software  specifications 
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4C.  Perform  software  testing  forquality  assurance 
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5.  Conduct  research  study  using  screen-based  simulation 
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5A.  Obtain  IRS  approval  from  UCLA  and  USAMRMC  HRPO 
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SB.  Recruit  subjects  to  test  screen-based  simulation 
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SC.  Conduct  follow  up  Interviews  with  select  subjects 
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23-27(10/2018) 

n 

3 

5D.  Perform  quantitative  and  qualitative  data  analysis 
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25-27(10/2018) 

SE.  Prepare  and  deliver  final  report  (Deliverable) 
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25-30(01/2019) 
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5 

6.  Write  manuscript  for  publication  (Deliverable) 
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Accomplishment:  Since  the  last  submission  on  4/14/2017,  we  have  made 
progress  in  the  development  of  our  screen-based  simulation  system; 
finalizing  the  domain  ontology  and  establishing  learning  objectives  and 
sequence  of  events  for  our  first  scenario.  We  have  also  completed  the 
team  training  literature  review  and  SME  and  focus  group  interviews. 

Goals/Milestones 

CY16  Goal  -  Gap  analysis 

IZI  Literature  and  video  review 

IZI  Interview  subject  matter  experts  and  focus  groups 

CY17  Goals  -  Complete  screen-based  simulation 

□  Complete  domain  ontology  and  study  scenarios 

□  Complete  framework  design  for  online  team  training  and  assessment 

□  Complete  programming  of  the  interactive  environment 

□  Recruit  subjects  to  test  screen-based  simulation 
CY18  Goal  -  Complete  research  data  collection 

□  Continue  to  recruit  subjects 

□  Perform  qualitative  and  quantitative  data  analysis 
Comments/Challenges/lssues/Concerns: 

Encountered  delays  in  programming  and  scenario  development. 
Timeline  adjusted  with  6-month  no-cost  extension  (request  in  progress). 
Budget  Expenditure  to  Date 
Projected  Expenditure:  $1 ,148,000 
Actual  Expenditure:  $478,559.19 
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